Multimodal image and audio music transcription

Abstract Optical Music Recognition (OMR) and Automatic Transcription (AMT) stand for the research fields that aim at obtaining a structured digital representation from sheet music images acoustic recordings, respectively. While these have traditionally evolved independently, fact both tasks may share same output poses question of whether they could be combined in synergistic manner to exploit individual transcription advantages depicted by each modality. To evaluate this hypothesis, paper presents multimodal framework combines predictions two neural end-to-end OMR AMT systems considering local alignment approach. We assess several experimental scenarios with monophonic pieces our approach under different conditions systems. In general, clearly outperforms single recognition modalities, attaining relative improvement close $$40\%$$ 40 % best case. Our initial premise is, therefore, validated, thus opening avenues further OMR-AMT transcription.